Many computational problems can be subdivided into independent or loosely-dependent tasks, which can be distributed among a group of processors or systems and executed in parallel. This often permits the main problem to be solved faster than would be possible if all the tasks were performed by a single processor or system. Cooperating processors and systems can be coordinated as necessary by transmitting messages between them. Messages can also be used to distribute work and to collect results. Some partitionings or decompositions of problems can place significant demands on a message passing infrastructure, either by sending and receiving a large number of messages, or by transferring large amounts of data within the messages.
Messages may be transferred over a number of different communication channels, or fabrics. For example, processors executing on the same physical machine may be able to communicate efficiently using shared memory, while processors on different machines may communicate through a high-speed network.
To prevent the varying operational requirements of these different communication fabrics from causing extra complexity in message-passing applications, a standard set of message passing functions may be defined to perform the standard functions over each type of fabric. One standard library definition is the Message Passing Interface (“MPI”) from the members of the MPI Forum. An MPI library may provide the standard functions over one or more fabrics. However, as the number of fabrics supported by a library increases, the message passing performance tends to decrease.
An issue that occurs in high process count jobs is the necessity to establish and maintain a high number of point-to-point connections between the processes of the parallel job. Dynamic (or lazy) connection establishment is typically used to avoid a massive up-front connection establishment phase by delaying the connection establishment until two processes start to communicate.
However, the potential need for processing a connection request at any time during job execution imposes additional stress on a progress engine: every once in a while, all connections need to be queried, and new connections established, if low-level network timeouts are to be avoided. The overhead associated with connection establishment and querying of a growing number of virtual channels naturally slows down data transfer of the progress engine. This situation is especially pronounced when a posted receive queue includes requests with a MPI_ANY_SOURCE process selector, because this requires all fabrics and virtual channels to be queried.